3D time-of-flight (ToF) cameras acquire depth images by determining the time which radiation, preferably light, needs from a source to an object and back to the camera. This is generally done by illuminating the scene discontinuously and applying a convolution of a temporal window (strictly speaking: a sequence of windows) to the backscattered incident optical signal. Typically, three quantities are unknown and have to be determined for each pixel individually: the object's distance, its reflectivity and the intensity of ambient light. Therefore, one or more (dependent on the number of unknowns) measurements, for instance at least three measurements in case of three unknowns, are necessary to determine these unknowns.
There are several reasons, why current ToF cameras and methods do not deliver optimal depth maps for moving sceneries. One reason is the motion blur affecting each raw image. Because the ToF camera is integrating the incident signal over a certain time window, edges and fine details of moving objects are blurred. Another reason is the temporal delay between raw data acquisitions. Current ToF cameras are not able to acquire all raw images instantaneously, but have to capture them consecutively. There are ToF cameras which acquire subsets of necessary raw values (called subframes) in parallel, but currently no ToF camera exists which measures all raw values synchronously.
If one or multiple of the unknowns (depth, background light, reflectivity) change during that process, the reconstruction generates incorrect results. More precisely, if at least one of the three unknowns (depth, background light, reflectivity) changes, the computed depth of affected pixels is incorrect. It should be noted that also other data channels generated by the ToF camera, e.g. describing the measured intensity of non-modulated radiation or the modulation amplitude of the detected signal (typical for continuous-wave ToF systems), will contain corrupt data in that case. In the following, the explanation will focus on the computed depth, but the argumentation and the proposed methods and devices are also valid and equally applicable for all other processed channels.
These kinds of errors might be caused, for instance, by rapidly changing features of the scene, for instance by moving depth or color edges. If the movement is parallel to the projection beam of a certain pixel, the signal deviations affecting this pixel are small due to the typically low speed of the objects. In contrast, laterally moving edges effect rapid changes of raw values, thus leading to strong errors and strong motion artifacts. When regarding the temporal signal of one raw channel of one pixel, a discontinuity occurs at the time step as the edge hits the pixel.
One method to prevent motion artifacts using continuous-wave ToF cameras is known from Lindner, Kolb: Compensation of Motion Artifacts for Time-of-Flight Cameras, Dynamic 3D Imaging, LNCS 5742/2009, p. 16-27, DOI 10.1007/978-3-642-03778-8, Berlin/Heidelberg 2009. Their approach performs optical flow estimation on all raw images to measure the movements of regions in the scene. In a further step, the estimated flow fields are used to warp and register the raw images. This ensures that each depth value is constructed from raw data describing the same point in the scene, and thus prevents motion artifacts. The drawback of this approach is, however, the high computational effort, which necessitates means like GPU implementations to process the data with appropriate speed. Further drawbacks are the necessity for intensity normalization of different raw channels and that there is no dense reconstruction (necessity for inpainting).
A Continuous-wave ToF sensor (PMD sensor) is described in Schwarte, R., Heinol, H. G., Xu, Z., Hartmann, K.: New active 3D vision system based on rf-modulation interferometry of incoherent light, in Casasent, D. P. (ed.) Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, vol. 2588, pp. 126-134 (1995) and in Spirig, T., Seitz, P., Heitger, F.: The lock-in CCD. Two-dimensional synchronous detection of light. IEEE J. Quantum Electronics 31, 1705-1708 (1995).
More information about the general technology of TOF cameras can be found in Elkhalili, O., Schrey, O., Ulfig, W., Brockherde, W., Hosticka, B. J., Mengel, P., Listl, L.: A 64×8 pixel 3-D CMOS time-of flight image sensor for car safety applications (2006), in Gokturk, S. B., Yalcin, H., Bamji, C.: A time-of-flight depth sensor—System description, issues and solutions, in http://www.canesta.com/assets/pdf/technicalpapers/CVPR_Submission_TOF.pdf, and in Oggier, T., Lehmann, M., Kaufmann, R., Schweizer, M., Richter, M., Metzler, P., Lang, G., Lustenberger, F., Blanc, N.: An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (2004), and in Ringbeck, T., Hagebeuker, B.: A 3D time-of-flight camera for object detection (2007).